Update bash ex-scripts to be linter compliant #4151

WalterKolczynski-NOAA · 2025-10-15T11:53:08Z

Description

Updates the ex-scripts to be compliant with both shellcheck and shfmt. This does not necessarily enforce some standards not checked by shellcheck. Export statements are added to some variables to avoid 'not used' errors. In the future, a more careful analysis should be done to see if these variables are used downstream by executables or if they are truly unused, in which case they should be removed.

Scripts employing MPMD are also updated to use run_mpmd.sh. This resulted in some retooling of those scripts.

Refs: #397

Type of change

Bug fix (fixes something broken)
New feature (adds functionality)
Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO
Does this change require an update to any of the following submodules? NO

How has this been tested?

Full test suite on Ursa

Checklist

Exports variables reported by shellcheck as being unused. A more careful examination is needed to remove unnecessary variables.

JessicaMeixner-NOAA

Can someone point me to where 4 spaces instead of 2 is the standard for the g-w or some linter check? I was unaware that was a thing, since 2 spaces is very widely used across the g-w.

This PR needs to be tested on WCOSS2 to ensure that all wave AWIPS, etc jobs are still running as expected as this changes those scripts and I think most of the changes were spaces, but I'm still going through to confirm as github isn't showing all of them as such. Additionally, wave jobs should be checked that the contents is the same before and after this PR as there were a few places where commands were changed.

scripts/exgfs_wave_post_pnt.sh

scripts/exgfs_wave_prdgen_bulls.sh

DavidHuber-NOAA · 2025-10-15T12:58:19Z

Can someone point me to where 4 spaces instead of 2 is the standard for the g-w or some linter check? I was unaware that was a thing, since 2 spaces is very widely used across the g-w.

@WalterKolczynski-NOAA I think that this comes from the change you made to .editorconfig:
https://github.com/NOAA-EMC/global-workflow/blame/5fa67e3339e300c9d380e32c98dba541948fbfcc/.editorconfig#L25-L38

JessicaMeixner-NOAA · 2025-10-15T13:31:37Z

Can someone point me to where 4 spaces instead of 2 is the standard for the g-w or some linter check? I was unaware that was a thing, since 2 spaces is very widely used across the g-w.

@WalterKolczynski-NOAA I think that this comes from the change you made to .editorconfig: https://github.com/NOAA-EMC/global-workflow/blame/5fa67e3339e300c9d380e32c98dba541948fbfcc/.editorconfig#L25-L38

So this is a new thing. This even seems inconsistent in .editorconfig as well. Is .editorconfig where we should be looking for these sorts of standards in the future?

WalterKolczynski-NOAA · 2025-10-15T14:00:49Z

Can someone point me to where 4 spaces instead of 2 is the standard for the g-w or some linter check? I was unaware that was a thing, since 2 spaces is very widely used across the g-w.

@WalterKolczynski-NOAA I think that this comes from the change you made to .editorconfig: https://github.com/NOAA-EMC/global-workflow/blame/5fa67e3339e300c9d380e32c98dba541948fbfcc/.editorconfig#L25-L38

So this is a new thing. This even seems inconsistent in .editorconfig as well. Is .editorconfig where we should be looking for these sorts of standards in the future?

Yes. I will note before this point, no standard was enforced for bash scripts, which is why different scripts had different indentation. This PR is part of applying a standard to all the legacy scripts so enforcement can begin. Four spaces is a compromise between space usage and visibility, and is consistent with the python scripts.

DavidHuber-NOAA · 2025-11-12T12:50:01Z

scripts/exgdas_atmos_chgres_forenkf.sh

-if [ $CFP_MP = "YES" ]; then
-    nm=0
-fi


Is this set at the JJob level?

It's not used anymore with the switch to using run_mpmd.sh.

DavidHuber-NOAA · 2025-11-12T12:56:41Z

scripts/exgdas_atmos_verfozn.sh

+    for filenc4 in diag*nc4.gz; do
+        netcdf=1
+        file=$(echo "${filenc4}" | cut -d'.' -f1-2).gz
+        mv "${filenc4}" "${file}"
+    done


This needs a file existence check. If there are no diag*nc4.gz files, then filenc4 will be assigned the literal diag*nc4.gz.

Actually, I think this should be reverted to the previous logic. There are some variables that could be removed (data_available, for instance), but otherwise I don't think this script should change logic.

Shellcheck doesn't like looping over ls: https://www.shellcheck.net/wiki/SC2045

Recommend we turn on shopt -s nullglob (as noted on that page) anyway for all our code to prevent * being treated as literal anywhere.

aerorahul

looks ok to me. just a few comments.
thanks for this. Next, we need a github action that prevents code from getting in that does not conform to the standards spelled out in editorconfig.

scripts/exglobal_atmos_ensstat.sh

scripts/exgfs_prdgen_manager.sh

DavidHuber-NOAA · 2025-11-14T19:04:33Z

scripts/exglobal_atmos_sfcanl.sh

-NTHREADS_CYCLE=${NTHREADS_CYCLE:-24}
-APRUN_CYCLE=${APRUN_CYCLE:-${APRUN:-""}}
+CYCLESH="${CYCLESH:-${USHgfs}/global_cycle.sh}"
+REGRIDSH="${REGRIDSH:-${USHgfs}/regrid_gsiSfcIncr_to_tile.sh}"


This seems like an inconsistency. Sometimes, when a variables is defined with a fallback option, the fallback is quoted (e.g. https://github.com/NOAA-EMC/global-workflow/pull/4151/files#diff-4cc7890265f85821b79705d4a49753b8bfbdcdd1003129c2a1052ef5ee07542aR49). Other times (like here), the entire RHS is quoted. Is one approach preferred?

DavidHuber-NOAA · 2025-11-14T19:05:55Z

scripts/exglobal_atmos_tropcy_qc_reloc.sh


 # Make sure we are in the $DATA directory
-cd "${DATA}"
+cd "${DATA}" || exit


Should probably be exit 1 so the J-Job handles the failure correctly.

DavidHuber-NOAA · 2025-11-14T19:31:03Z

scripts/exglobal_atmos_products.sh

-       last=${ncount}
+    # Each set represents a group of files
+    if [[ "${nset}" == 1 ]]; then
+        grp="a" # TODO: this should be "a" when we eventually rename the pressure grib2 files per EE2 convention


This work has been completed

Suggested change

grp="a" # TODO: this should be "a" when we eventually rename the pressure grib2 files per EE2 convention

grp="a"

DavidHuber-NOAA · 2025-11-14T19:33:44Z

scripts/exglobal_atmos_products.sh

-              -grib "${COMOUT_ATMOS_GRIB_0p25}/${PREFIX}wgne.${fhr3}.grib2"
-  fi
+    grp="a"
+    if [[ "${FORECAST_HOUR}" -gt 0 && "${FORECAST_HOUR}" -le ${FHMAX_WGNE:-0} ]]; then


Just a note that double-quotes are not needed when doing arithmetic logic. No need to change.

DavidHuber-NOAA · 2025-11-14T19:36:17Z

scripts/exglobal_atmos_tropcy_qc_reloc.sh

 tmmark_uc=${tmmark^^}

-iflag=0
+export iflag=0


No need to export this. iflag isn't used upstream.

Suggested change

export iflag=0

iflag=0

DavidHuber-NOAA · 2025-11-14T19:40:29Z

scripts/exglobal_diag.sh

+    diagtype[0]="conv conv_gps conv_ps conv_pw conv_q conv_sst conv_t conv_tcp conv_uv conv_spd"
+    diagtype[1]="pcp_ssmi_dmsp pcp_tmi_trmm"
+    diagtype[2]="sbuv2_n16 sbuv2_n17 sbuv2_n18 sbuv2_n19 gome_metop-a gome_metop-b omi_aura mls30_aura ompsnp_npp ompstc8_npp  ompstc8_n20 ompsnp_n20 ompstc8_n21 ompsnp_n21 ompslp_npp gome_metop-c"
+    diagtype[3]="msu_n14 sndr_g08 sndr_g11 sndr_g12 sndr_g13 sndr_g08_prep sndr_g11_prep sndr_g12_prep sndr_g13_prep sndrd1_g11 sndrd2_g11 sndrd3_g11 sndrd4_g11 sndrd1_g12 sndrd2_g12 sndrd3_g12 sndrd4_g12 sndrd1_g13 sndrd2_g13 sndrd3_g13 sndrd4_g13 sndrd1_g14 sndrd2_g14 sndrd3_g14 sndrd4_g14 sndrd1_g15 sndrd2_g15 sndrd3_g15 sndrd4_g15 amsua_n15 amsua_n16 amsua_n17 amsub_n15 amsub_n16 amsub_n17 hsb_aqua airs_aqua amsua_aqua imgr_g08 imgr_g11 imgr_g12 imgr_g14 imgr_g15 ssmi_f13 ssmi_f15 amsua_n18 amsua_metop-a mhs_n18 mhs_metop-a amsre_low_aqua amsre_mid_aqua amsre_hig_aqua ssmis_f16 ssmis_f17 ssmis_f18 ssmis_f19 ssmis_f20 iasi_metop-a amsua_n19 mhs_n19 seviri_m08 seviri_m09 seviri_m10 seviri_m11 cris_npp cris-fsr_npp cris-fsr_n20 atms_npp atms_n20 amsua_metop-b mhs_metop-b iasi_metop-b avhrr_metop-b avhrr_n18 avhrr_n19 avhrr_metop-a amsr2_gcom-w1 gmi_gpm saphir_meghat ahi_himawari8 abi_g16 abi_g17 amsua_metop-c mhs_metop-c iasi_metop-c avhrr_metop-c viirs-m_npp viirs-m_j1 abi_g18 ahi_himawari9 viirs-m_j2 cris-fsr_n21 atms_n21 abi_g19"


This was probably duplicated with a merge.

Suggested change

diagtype[0]="conv conv_gps conv_ps conv_pw conv_q conv_sst conv_t conv_tcp conv_uv conv_spd"

diagtype[1]="pcp_ssmi_dmsp pcp_tmi_trmm"

diagtype[2]="sbuv2_n16 sbuv2_n17 sbuv2_n18 sbuv2_n19 gome_metop-a gome_metop-b omi_aura mls30_aura ompsnp_npp ompstc8_npp ompstc8_n20 ompsnp_n20 ompstc8_n21 ompsnp_n21 ompslp_npp gome_metop-c"

diagtype[3]="msu_n14 sndr_g08 sndr_g11 sndr_g12 sndr_g13 sndr_g08_prep sndr_g11_prep sndr_g12_prep sndr_g13_prep sndrd1_g11 sndrd2_g11 sndrd3_g11 sndrd4_g11 sndrd1_g12 sndrd2_g12 sndrd3_g12 sndrd4_g12 sndrd1_g13 sndrd2_g13 sndrd3_g13 sndrd4_g13 sndrd1_g14 sndrd2_g14 sndrd3_g14 sndrd4_g14 sndrd1_g15 sndrd2_g15 sndrd3_g15 sndrd4_g15 amsua_n15 amsua_n16 amsua_n17 amsub_n15 amsub_n16 amsub_n17 hsb_aqua airs_aqua amsua_aqua imgr_g08 imgr_g11 imgr_g12 imgr_g14 imgr_g15 ssmi_f13 ssmi_f15 amsua_n18 amsua_metop-a mhs_n18 mhs_metop-a amsre_low_aqua amsre_mid_aqua amsre_hig_aqua ssmis_f16 ssmis_f17 ssmis_f18 ssmis_f19 ssmis_f20 iasi_metop-a amsua_n19 mhs_n19 seviri_m08 seviri_m09 seviri_m10 seviri_m11 cris_npp cris-fsr_npp cris-fsr_n20 atms_npp atms_n20 amsua_metop-b mhs_metop-b iasi_metop-b avhrr_metop-b avhrr_n18 avhrr_n19 avhrr_metop-a amsr2_gcom-w1 gmi_gpm saphir_meghat ahi_himawari8 abi_g16 abi_g17 amsua_metop-c mhs_metop-c iasi_metop-c avhrr_metop-c viirs-m_npp viirs-m_j1 abi_g18 ahi_himawari9 viirs-m_j2 cris-fsr_n21 atms_n21 abi_g19"

DavidHuber-NOAA · 2025-11-14T19:55:46Z

scripts/exglobal_atmos_analysis.sh

+    # shellcheck disable=SC2153
+    "${USHgfs}/create_gsi_info.sh" sat "${PDY}${cyc}" "${DATA}"


What variable(s) did shellcheck think were misspelled here?

DavidHuber-NOAA

Overall this is a great changeset! Thanks Walter. There's just a couple of minor issues that need addressing and then we can start testing.

WalterKolczynski-NOAA added 7 commits October 11, 2025 00:54

Update exscripts for shellcheck except unused variables [SC2034]

b0ee5dd

Apply shfmt to exscripts

f4e93b5

Remove unneeded APRUNCFP in anlcalc

85357db

Export "unused" variables

3f23e96

Exports variables reported by shellcheck as being unused. A more careful examination is needed to remove unnecessary variables.

Fix errors introduced during formatting updates

c658195

Fix error in skew-t script

d39104f

Shfmt skew-T

51d49d3

JessicaMeixner-NOAA reviewed Oct 15, 2025

View reviewed changes

scripts/exgfs_wave_post_pnt.sh Show resolved Hide resolved

scripts/exgfs_wave_prdgen_bulls.sh Show resolved Hide resolved

TerrenceMcGuinness-NOAA self-requested a review October 15, 2025 12:54

WalterKolczynski-NOAA added 5 commits October 17, 2025 11:06

Merge develop into feature/shfmt_exscripts

f1743c0

Merge develop' into feature/shfmt_exscripts

61419d3

Merge branch 'develop' into feature/shfmt_exscripts

7799ca5

Apply shfmt to analysis changes

cd4f1ea

Merge branch 'develop' into feature/shfmt_exscripts

006ab6c

WalterKolczynski-NOAA added 4 commits November 4, 2025 20:17

Merge origin/develop into feature/shfmt_exscripts

407affd

Merge develop into feature/shfmt_exscripts

e19b80e

Merge develop into feature/shfmt_exscripts

c7e91f2

Merge develop into feature/shfmt_exscripts

f59183c

DavidHuber-NOAA reviewed Nov 12, 2025

View reviewed changes

aerorahul reviewed Nov 12, 2025

View reviewed changes

scripts/exglobal_atmos_ensstat.sh Outdated Show resolved Hide resolved

scripts/exglobal_atmos_ensstat.sh Outdated Show resolved Hide resolved

scripts/exgfs_prdgen_manager.sh Outdated Show resolved Hide resolved

WalterKolczynski-NOAA added 4 commits November 13, 2025 10:24

Make minor updates based on PR review

0cf2240

Make missed correction based on review comment

3f56b21

Merge develop into feature/shfmt_exscripts

c06d9b1

Update j-jobs that have fallen out of compliance

3acb9fc

DavidHuber-NOAA reviewed Nov 14, 2025

View reviewed changes

DavidHuber-NOAA requested changes Nov 14, 2025

View reviewed changes

	grp="a" # TODO: this should be "a" when we eventually rename the pressure grib2 files per EE2 convention
	grp="a"

		# shellcheck disable=SC2153
		"${USHgfs}/create_gsi_info.sh" sat "${PDY}${cyc}" "${DATA}"

Update bash ex-scripts to be linter compliant #4151

Are you sure you want to change the base?

Update bash ex-scripts to be linter compliant #4151

Uh oh!

Conversation

WalterKolczynski-NOAA commented Oct 15, 2025

Description

Type of change

Change characteristics

How has this been tested?

Checklist

Uh oh!

JessicaMeixner-NOAA left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

DavidHuber-NOAA commented Oct 15, 2025

Uh oh!

JessicaMeixner-NOAA commented Oct 15, 2025

Uh oh!

WalterKolczynski-NOAA commented Oct 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

WalterKolczynski-NOAA Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aerorahul left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DavidHuber-NOAA left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

WalterKolczynski-NOAA commented Oct 15, 2025 •

edited

Loading

WalterKolczynski-NOAA Nov 13, 2025 •

edited

Loading